Logi.Tude is a famous game developer from Finland founded in 2008, which offers a comprehensive online gaming platform with more than 15 million customers. The platform currently manages three major online real-time strategy games. Revenues are generated either through in-game purchases for avatars and mods or through a monthly subscription fee that may vary depending on which services and games a customer is using.
While Logi.Tude focused on platform development within the last five years by introducing new games and extending the current listings by introducing mods and unique games. The platform enjoyed substantial growth right after its launch. However, within the last 18 months, Logi.Tude saw more and more customers becoming inactive and canceling subscriptions. Re-activating these lost customers turns out to consume substantial resources and threatens Logi.Tude’s cash flow that is urgently needed to secure the development of future games. Therefore, the board of directors calls in an emergency meeting to discuss measures. Dobs Bavis, the freshly appointed CMO responsible for customer relations, has difficulty explaining the current developments and is facing harsh criticism. “We cannot accept that we do not have a clear understanding of why customers are leaving us. Until we develop a better understanding and can take measures, we, however, expect marketing to identify customers who may potentially leave the platform and to address these customers with adequate retention measures,” requests Logi.Tude founder and current CEO Charles Selenski.
Right after the board meeting, Dobs contacts you as the head of the data science department and asks you to conduct a project that
can predict which customers are likely to leave the company
understands from the available CRM database which factors help best predict customer churn.
To achieve this, Dob provides you with the following data set from the CRM system.
import pandas as pd
# Creating the table_desc DataFrame
table_desc = pd.DataFrame({
'CRM_Variable': ["customer_id", "churn", "monthly_bill", "gender", "millennial", "clanmembership", "platformsince", "avatar", "internet_connect", "extrapro_account", "contract_length", "customer_support", "payment_type"],
'Description': ["unique identifier for each customer", "information if customer left company or stayed with service", "payment data from customer based on returns from subscription and in-game purchases", "customer gender", "categorical information about age", "Information if customer is a member of a gaming clan", "time since the customer is active on the platform", "information if the customer has bought a customized avatar", "average speed of internet connection", "information if the customer is subscribing to a pro gaming account with bonus experience", "type of subscription model", "Information if the customer has been in contact with support within the last 6 months", "information on how the customer pays invoices"],
'Measurement': ["unique numerical identifier", "Yes/No", "spending in €", "male/female", "Millennial or Gen Z", "Yes/No", "number of months", "Yes/No", "DSL, DSL16, Fiber", "Yes/No", "month-to-month, 1 year, 2 years", "Yes/No", "Pre-Paid, Electronic Check, Bank Transfer, Credit Card"]
})
# Printing the table in HTML format
from IPython.display import display, HTML
html_table = table_desc.to_html()
display(HTML(f"<caption>Table 8.1: Logi.Tude's CRM Data</caption>{html_table}"))
| CRM_Variable | Description | Measurement | |
|---|---|---|---|
| 0 | customer_id | unique identifier for each customer | unique numerical identifier |
| 1 | churn | information if customer left company or stayed with service | Yes/No |
| 2 | monthly_bill | payment data from customer based on returns from subscription and in-game purchases | spending in € |
| 3 | gender | customer gender | male/female |
| 4 | millennial | categorical information about age | Millennial or Gen Z |
| 5 | clanmembership | Information if customer is a member of a gaming clan | Yes/No |
| 6 | platformsince | time since the customer is active on the platform | number of months |
| 7 | avatar | information if the customer has bought a customized avatar | Yes/No |
| 8 | internet_connect | average speed of internet connection | DSL, DSL16, Fiber |
| 9 | extrapro_account | information if the customer is subscribing to a pro gaming account with bonus experience | Yes/No |
| 10 | contract_length | type of subscription model | month-to-month, 1 year, 2 years |
| 11 | customer_support | Information if the customer has been in contact with support within the last 6 months | Yes/No |
| 12 | payment_type | information on how the customer pays invoices | Pre-Paid, Electronic Check, Bank Transfer, Credit Card |
# Import Data
LogiTude_CRM_Data_train = pd.read_csv("Data/my_train_dataframe.csv")
LogiTude_CRM_Data_test = pd.read_csv("Data/my_test_dataframe.csv")
# Display the structure of the DataFrame
print(LogiTude_CRM_Data_train.info())
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10434 entries, 0 to 10433 Data columns (total 13 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 ...1 10434 non-null int64 1 churn 10434 non-null object 2 monthly_bill 10434 non-null float64 3 gender 10434 non-null object 4 millenial 10434 non-null object 5 clanmembership 10434 non-null object 6 platformsince 10434 non-null int64 7 avatar 10434 non-null object 8 internet_connect 10434 non-null object 9 extrapro_account 10434 non-null object 10 contract_length 10434 non-null object 11 customer_support 10434 non-null object 12 payment_type 10434 non-null object dtypes: float64(1), int64(2), object(10) memory usage: 1.0+ MB None
Most of the variables were imported as objects, as Python did not understand that these should be treated as categories so that we can easily include them in the analysis. We must thus re-code these variables.
# Recode all object columns to categorical
LogiTude_CRM_Data_train[LogiTude_CRM_Data_train.select_dtypes(include='object').columns] = LogiTude_CRM_Data_train.select_dtypes(include='object').astype('category')
LogiTude_CRM_Data_test[LogiTude_CRM_Data_test.select_dtypes(include='object').columns] = LogiTude_CRM_Data_test.select_dtypes(include='object').astype('category')
# Identify categorical columns with 2 or 3 unique values
categorical_columns_train = LogiTude_CRM_Data_train.select_dtypes(['category']).columns
selected_columns_train = [col for col in categorical_columns_train if LogiTude_CRM_Data_train[col].nunique() in [2, 3,4]]
categorical_columns_test = LogiTude_CRM_Data_test.select_dtypes(['category']).columns
selected_columns_test = [col for col in categorical_columns_test if LogiTude_CRM_Data_test[col].nunique() in [2, 3,4]]
from sklearn.preprocessing import OneHotEncoder
# One-hot encode selected categorical columns
encoder = OneHotEncoder(sparse=False, drop='first')
encoded_columns_train = pd.DataFrame(encoder.fit_transform(LogiTude_CRM_Data_train[selected_columns_train]))
encoded_columns_train.columns = encoder.get_feature_names_out(selected_columns_train)
encoded_columns_test = pd.DataFrame(encoder.fit_transform(LogiTude_CRM_Data_test[selected_columns_test]))
encoded_columns_test.columns = encoder.get_feature_names_out(selected_columns_test)
/Users/rhiannegonsalves/opt/anaconda3/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. warnings.warn( /Users/rhiannegonsalves/opt/anaconda3/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. warnings.warn(
# Concatenate the one-hot encoded columns with the original DataFrame
Logi_Data_encoded_train = pd.concat([LogiTude_CRM_Data_train, encoded_columns_train], axis=1)
Logi_Data_encoded_test = pd.concat([LogiTude_CRM_Data_test, encoded_columns_test], axis=1)
# Change column names
Logi_Data_encoded_train['internet_connect_DSL_16MB_S'] = Logi_Data_encoded_train['internet_connect_DSL_16MB/S']
Logi_Data_encoded_train['internet_connect_Fiber_optic'] = Logi_Data_encoded_train['internet_connect_Fiber optic']
Logi_Data_encoded_train['contract_length_One_year'] = Logi_Data_encoded_train['contract_length_One year']
Logi_Data_encoded_train['contract_length_Two_year'] = Logi_Data_encoded_train['contract_length_Two year']
Logi_Data_encoded_train['payment_type_Credit_card'] = Logi_Data_encoded_train['payment_type_Credit card (automatic)']
Logi_Data_encoded_train['payment_type_Electronic_check'] = Logi_Data_encoded_train['payment_type_Electronic check']
old_columns = ['contract_length_One year', 'contract_length_Two year', 'internet_connect_DSL_16MB/S', 'internet_connect_Fiber optic', 'payment_type_Credit card (automatic)', 'payment_type_Electronic check']
Logi_Data_encoded_train = Logi_Data_encoded_train.drop(columns=old_columns)
# Display the updated DataFrame
print(Logi_Data_encoded_train.head())
...1 churn monthly_bill gender millenial clanmembership platformsince \ 0 12312 No 59.65 Male Gen Z Yes 15 1 6234 No 20.70 Male Gen Z No 3 2 12798 No 25.45 Female Gen Z Yes 71 3 702 No 83.15 Male Gen Z No 35 4 6227 No 25.75 Male Gen Z No 65 avatar internet_connect extrapro_account ... avatar_Yes \ 0 Yes DSL_16MB/S Yes ... 1.0 1 Yes Base DSL_16MB/S No ... 1.0 2 Yes Base DSL_16MB/S Yes ... 1.0 3 Yes Fiber optic Yes ... 1.0 4 Yes Base DSL_16MB/S Yes ... 1.0 extrapro_account_Yes customer_support_Yes payment_type_PrePaid \ 0 1.0 1.0 0.0 1 0.0 0.0 1.0 2 1.0 0.0 0.0 3 1.0 0.0 0.0 4 1.0 0.0 1.0 internet_connect_DSL_16MB_S internet_connect_Fiber_optic \ 0 1.0 0.0 1 0.0 0.0 2 0.0 0.0 3 0.0 1.0 4 0.0 0.0 contract_length_One_year contract_length_Two_year \ 0 0.0 0.0 1 0.0 0.0 2 0.0 1.0 3 0.0 0.0 4 0.0 1.0 payment_type_Credit_card payment_type_Electronic_check 0 0.0 0.0 1 0.0 0.0 2 0.0 0.0 3 0.0 1.0 4 0.0 0.0 [5 rows x 27 columns]
Logi_Data_encoded_test['internet_connect_DSL_16MB_S'] = Logi_Data_encoded_test['internet_connect_DSL_16MB/S']
Logi_Data_encoded_test['internet_connect_Fiber_optic'] = Logi_Data_encoded_test['internet_connect_Fiber optic']
Logi_Data_encoded_test['contract_length_One_year'] = Logi_Data_encoded_test['contract_length_One year']
Logi_Data_encoded_test['contract_length_Two_year'] = Logi_Data_encoded_test['contract_length_Two year']
Logi_Data_encoded_test['payment_type_Credit_card'] = Logi_Data_encoded_test['payment_type_Credit card (automatic)']
Logi_Data_encoded_test['payment_type_Electronic_check'] = Logi_Data_encoded_test['payment_type_Electronic check']
old_columns = ['contract_length_One year', 'contract_length_Two year', 'internet_connect_DSL_16MB/S', 'internet_connect_Fiber optic', 'payment_type_Credit card (automatic)', 'payment_type_Electronic check']
Logi_Data_encoded_test = Logi_Data_encoded_test.drop(columns=old_columns)
# Display the updated DataFrame
print(Logi_Data_encoded_test.head())
...1 churn monthly_bill gender millenial clanmembership platformsince \ 0 6 Yes 99.65 Female Gen Z No 8 1 8 No 29.75 Female Gen Z No 10 2 12 No 18.95 Male Gen Z No 16 3 16 No 113.25 Female Gen Z Yes 69 4 24 No 59.90 Female Gen Z Yes 58 avatar internet_connect extrapro_account ... avatar_Yes \ 0 Yes Fiber optic Yes ... 1.0 1 No DSL_16MB/S No ... 0.0 2 Yes Base DSL_16MB/S No ... 1.0 3 Yes Fiber optic Yes ... 1.0 4 Yes DSL_16MB/S Yes ... 1.0 extrapro_account_Yes customer_support_Yes payment_type_PrePaid \ 0 1.0 1.0 0.0 1 0.0 0.0 1.0 2 0.0 0.0 0.0 3 1.0 0.0 0.0 4 1.0 1.0 0.0 internet_connect_DSL_16MB_S internet_connect_Fiber_optic \ 0 0.0 1.0 1 1.0 0.0 2 0.0 0.0 3 0.0 1.0 4 1.0 0.0 contract_length_One_year contract_length_Two_year \ 0 0.0 0.0 1 0.0 0.0 2 0.0 1.0 3 0.0 1.0 4 0.0 1.0 payment_type_Credit_card payment_type_Electronic_check 0 0.0 1.0 1 0.0 0.0 2 1.0 0.0 3 1.0 0.0 4 1.0 0.0 [5 rows x 27 columns]
With the following code we run our Logistic Regression model. As you can see instead of lm we rely on the glm function from the sm library in Python that calls a generalized linear model. We hand in the dependent variable that we seperate again with ~ from the independent variables. With the addition of family "binomial" we tell Python that we want to execute a logistic regression with a binary dependent variable. The summary command finally provides us with the output.
import statsmodels.api as sm
formula = "churn_Yes ~ monthly_bill + gender_Male + millenial_Millenial + clanmembership_Yes + platformsince + avatar_Yes + internet_connect_DSL_16MB_S + internet_connect_Fiber_optic + extrapro_account_Yes + contract_length_One_year + contract_length_Two_year + customer_support_Yes + payment_type_Electronic_check + payment_type_Credit_card + payment_type_PrePaid "
# Fit logistic regression model
LR_Logi_Tude = sm.formula.glm(formula=formula, data=Logi_Data_encoded_train, family=sm.families.Binomial()).fit()
# Display summary
print(LR_Logi_Tude.summary())
# Access the coefficient of the intercept
intercept_coefficient = LR_Logi_Tude.params['Intercept']
# Display the coefficient of the intercept
print("Coefficient of the intercept:", intercept_coefficient)
# Access specific values
print("Null Deviance:", LR_Logi_Tude.null_deviance)
print("Residual Deviance:", LR_Logi_Tude.deviance)
print("AIC:", LR_Logi_Tude.aic)
Generalized Linear Model Regression Results
==============================================================================
Dep. Variable: churn_Yes No. Observations: 10434
Model: GLM Df Residuals: 10418
Model Family: Binomial Df Model: 15
Link Function: Logit Scale: 1.0000
Method: IRLS Log-Likelihood: -4450.4
Date: Tue, 02 Jan 2024 Deviance: 8900.9
Time: 01:21:04 Pearson chi2: 1.05e+04
No. Iterations: 7 Pseudo R-squ. (CS): 0.2650
Covariance Type: nonrobust
=================================================================================================
coef std err z P>|z| [0.025 0.975]
-------------------------------------------------------------------------------------------------
Intercept -0.9201 0.156 -5.891 0.000 -1.226 -0.614
monthly_bill 0.0097 0.003 3.227 0.001 0.004 0.016
gender_Male -0.0035 0.052 -0.067 0.946 -0.106 0.099
millenial_Millenial 0.2559 0.067 3.818 0.000 0.125 0.387
clanmembership_Yes -0.0889 0.057 -1.565 0.118 -0.200 0.022
platformsince -0.0356 0.002 -18.916 0.000 -0.039 -0.032
avatar_Yes -0.7691 0.117 -6.594 0.000 -0.998 -0.541
internet_connect_DSL_16MB_S 0.2639 0.150 1.757 0.079 -0.031 0.558
internet_connect_Fiber_optic 1.0729 0.215 4.985 0.000 0.651 1.495
extrapro_account_Yes 0.3106 0.066 4.733 0.000 0.182 0.439
contract_length_One_year -0.6954 0.086 -8.048 0.000 -0.865 -0.526
contract_length_Two_year -1.4485 0.135 -10.704 0.000 -1.714 -1.183
customer_support_Yes 0.3496 0.060 5.855 0.000 0.233 0.467
payment_type_Electronic_check 0.4157 0.076 5.471 0.000 0.267 0.565
payment_type_Credit_card 0.0124 0.092 0.136 0.892 -0.167 0.192
payment_type_PrePaid -0.0086 0.092 -0.094 0.925 -0.188 0.171
=================================================================================================
Coefficient of the intercept: -0.920125836168306
Null Deviance: 12112.984364582815
Residual Deviance: 8900.859262433474
AIC: 8932.859262433474
Strongly correlated variables within the model may cause multicollinearity and thus lead to inflated estimates. To control for multicollinearity we need to use the variance_inflation_factor function.
from sklearn.preprocessing import LabelEncoder
from statsmodels.stats.outliers_influence import variance_inflation_factor
# Encode categorical variables
label_encoder = LabelEncoder()
for col in ['gender', 'millenial', 'clanmembership', 'contract_length', 'customer_support', 'payment_type', 'internet_connect', 'avatar', 'extrapro_account']:
Logi_Data_encoded_train[col] = label_encoder.fit_transform(Logi_Data_encoded_train[col])
X = sm.add_constant(Logi_Data_encoded_train[['monthly_bill', 'gender', 'millenial', 'clanmembership', 'platformsince', 'avatar', 'extrapro_account', 'contract_length', 'customer_support', 'payment_type', 'internet_connect']])
# Calculate VIF
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
# Display the VIF values
print("Variance Inflation Factor (VIF):")
# Display VIF results
print("\nCheck for Multicollinearity\n")
print("Low Correlation:")
print(vif_data[vif_data["VIF"] <= 5])
print("\nHigh Correlation:\n")
print(vif_data[(vif_data["VIF"] > 5) ])
Variance Inflation Factor (VIF):
Check for Multicollinearity
Low Correlation:
Variable VIF
2 gender 1.000990
3 millenial 1.096444
4 clanmembership 1.174815
5 platformsince 2.653976
6 avatar 1.293505
7 extrapro_account 1.490612
8 contract_length 2.351154
9 customer_support 1.188234
10 payment_type 1.172621
High Correlation:
Variable VIF
0 const 21.786889
1 monthly_bill 10.452472
11 internet_connect 9.529729
As the output indicates, the two variables monthly_bill and internet_connect show high Variance Inflation Factors and thus point us on multicollinearity issues. Thus, we need to drop one of the two variables from the model.
Question: Which variable do you suggest to drop here? For which reason?
We can now build our final logistic regression model and control for one last time, that multicollinearity is no issue. We use the following code.
LR_Logi_Tude1 = sm.formula.glm(formula="churn_Yes ~ monthly_bill + gender_Male + millenial_Millenial + clanmembership_Yes + platformsince + avatar_Yes + extrapro_account_Yes + contract_length_One_year + contract_length_Two_year + customer_support_Yes + payment_type_Electronic_check + payment_type_Credit_card + payment_type_PrePaid",
data=Logi_Data_encoded_train,
family=sm.families.Binomial()).fit()
# Display summary
print(LR_Logi_Tude1.summary())
# Check for Multicollinearity
# Select relevant columns for VIF calculation
X1 = sm.add_constant(Logi_Data_encoded_train[['monthly_bill', 'gender', 'millenial', 'clanmembership', 'platformsince', 'avatar', 'extrapro_account', 'contract_length', 'customer_support', 'payment_type']])
# Calculate VIF
vif_data1 = pd.DataFrame()
vif_data1["Term"] = X1.columns
vif_data1["VIF"] = [variance_inflation_factor(X1.values, i) for i in range(X1.shape[1])]
# Display VIF results
print("\nCheck for Multicollinearity\n")
print("Low Correlation")
print(vif_data1[vif_data1["VIF"] <= 5])
print("\nModerate Correlation")
print(vif_data1[(vif_data1["VIF"] > 5) & (vif_data1["VIF"] <= 10)])
Generalized Linear Model Regression Results
==============================================================================
Dep. Variable: churn_Yes No. Observations: 10434
Model: GLM Df Residuals: 10420
Model Family: Binomial Df Model: 13
Link Function: Logit Scale: 1.0000
Method: IRLS Log-Likelihood: -4479.9
Date: Tue, 02 Jan 2024 Deviance: 8959.8
Time: 01:24:03 Pearson chi2: 1.07e+04
No. Iterations: 7 Pseudo R-squ. (CS): 0.2608
Covariance Type: nonrobust
=================================================================================================
coef std err z P>|z| [0.025 0.975]
-------------------------------------------------------------------------------------------------
Intercept -1.2094 0.124 -9.762 0.000 -1.452 -0.967
monthly_bill 0.0250 0.001 17.561 0.000 0.022 0.028
gender_Male -0.0014 0.052 -0.027 0.979 -0.104 0.101
millenial_Millenial 0.3065 0.066 4.615 0.000 0.176 0.437
clanmembership_Yes -0.0955 0.057 -1.690 0.091 -0.206 0.015
platformsince -0.0378 0.002 -20.646 0.000 -0.041 -0.034
avatar_Yes -0.7970 0.098 -8.114 0.000 -0.990 -0.604
extrapro_account_Yes 0.2662 0.064 4.142 0.000 0.140 0.392
contract_length_One_year -0.8040 0.085 -9.494 0.000 -0.970 -0.638
contract_length_Two_year -1.6034 0.133 -12.043 0.000 -1.864 -1.342
customer_support_Yes 0.3666 0.059 6.187 0.000 0.250 0.483
payment_type_Electronic_check 0.4413 0.076 5.837 0.000 0.293 0.589
payment_type_Credit_card 0.0074 0.091 0.081 0.935 -0.171 0.186
payment_type_PrePaid -0.0285 0.091 -0.313 0.754 -0.207 0.150
=================================================================================================
Check for Multicollinearity
Low Correlation
Term VIF
1 monthly_bill 1.671105
2 gender 1.000987
3 millenial 1.084389
4 clanmembership 1.173756
5 platformsince 2.567579
6 avatar 1.140830
7 extrapro_account 1.464332
8 contract_length 2.165749
9 customer_support 1.184120
10 payment_type 1.171911
Moderate Correlation
Empty DataFrame
Columns: [Term, VIF]
Index: []
As indicated by the VIF values, which are all well below three, we can outrule the presence of multicollinearity and thus starting with interpreting the model output.
Looking at the p-values, we can conclude that all variables but three show significant effects and thus influence the likelihood that a customer is leaving the platform. Only gender and credit card, and pre-paid payments do not significantly affect the churning likelihood.
For our categorical variables, we see that one category is always missing. In the case of contract length, e.g., monthly payments are not listed. This is because Python is treating the factors as dummy coded variables. One category is thus permanently excluded from the model. It serves as a reference category, which means that we can interpret the effects always in comparison to the missing reference category. In the case of contract length, we must thus conclude that longer contracts show a “lower” likelihood to churn, as indicated by the negative coefficients for the two categories, “One Year” and “Two Year” of our factor contract length.
Question: Go through all significant effects, interpret the direction of effect, and develop possible explanations for what you find.
Given the functional form of the Logit model, we can not simply interpret the coefficients from the logistic regression but need to transform them into the odds that someone is churning, given a change of each variable. We can ask Python to provide us with the Odds of each variable with this code.
import numpy as np
# Extract coefficients and confidence intervals
coefficients = LR_Logi_Tude1.params
conf_int = LR_Logi_Tude1.conf_int()
# Combine coefficients and confidence intervals
result = pd.concat([np.exp(coefficients), np.exp(conf_int)], axis=1)
result.columns = ['OR', 'CI_lower', 'CI_upper']
# Display the result
print(result)
OR CI_lower CI_upper Intercept 0.298388 0.234060 0.380395 monthly_bill 1.025364 1.022501 1.028234 gender_Male 0.998608 0.901475 1.106207 millenial_Millenial 1.358675 1.192852 1.547550 clanmembership_Yes 0.908895 0.813603 1.015348 platformsince 0.962933 0.959487 0.966392 avatar_Yes 0.450667 0.371741 0.546349 extrapro_account_Yes 1.304990 1.150542 1.480173 contract_length_One_year 0.447552 0.379109 0.528352 contract_length_Two_year 0.201215 0.155001 0.261208 customer_support_Yes 1.442822 1.284625 1.620501 payment_type_Electronic_check 1.554729 1.340608 1.803050 payment_type_Credit_card 1.007417 0.842609 1.204460 payment_type_PrePaid 0.971926 0.813185 1.161656
The Odds can now be interpreted as the corresponding likelihood of each variable for churning.
While the odds help us with understanding the impact of each variable on the churn likelihood we can not still make any predictions and are thus not able to judge the predictive power of our model. We must thus fit our model to the test data set and see how well we are able to predict churning customers. We achieve this with the following code that gives us the model's accuracy.
label_encoder = LabelEncoder()
for col in ['gender', 'millenial', 'clanmembership', 'contract_length', 'customer_support', 'payment_type', 'internet_connect']:
Logi_Data_encoded_test[col] = label_encoder.fit_transform(Logi_Data_encoded_test[col])
# Assuming Logi_Data_test is a numpy array or another data type
Logi_Data_test_df = pd.DataFrame(Logi_Data_encoded_test, columns=Logi_Data_encoded_train.columns)
# Predict probabilities
fitted_results_LR1 = LR_Logi_Tude1.predict(exog=Logi_Data_test_df)
# Convert to binary classes based on a threshold of 0.5
fitted_results_LR1_binary = (fitted_results_LR1 > 0.5).astype(int)
actual_values_numeric = np.where(Logi_Data_test_df['churn'].values == 'Yes', 1, 0)
# Calculate accuracy
misclassification_error = np.mean(fitted_results_LR1_binary != actual_values_numeric)
accuracy = 1 - misclassification_error
print(f"Logistic Regression Accuracy: {accuracy}")
Logistic Regression Accuracy: 0.8098888463012648
We can thus conclude that our model correctly predicts customer churn in 80.99% of all cases.
To grow a decision tree we need load DecisionTreeClassifierlibrary. We do this with the following command.
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt
from sklearn.tree import plot_tree
X_train = Logi_Data_encoded_train[['monthly_bill', 'gender', 'millenial', 'clanmembership', 'platformsince', 'avatar', 'internet_connect', 'extrapro_account', 'contract_length', 'customer_support', 'payment_type']]
y_train = Logi_Data_encoded_train['churn']
# Create the decision tree classifier with gini criterion
my_1sttree_LogiTude = DecisionTreeClassifier(criterion='gini', splitter='best', max_depth=7, min_samples_split=400, ccp_alpha=0.001)
# Fit the model to the training data
my_1sttree_LogiTude.fit(X_train, y_train)
# Plotting the tree
plt.figure(figsize=(20,10))
plot_tree(my_1sttree_LogiTude, filled=True, feature_names=X_train.columns.tolist())
plt.show()
We start our inspection at the root, where contract length is the first variable for a split. The tree shows that if you have a longer one- or two-year contract, you are immediately moved to the non-churning leave on the right that takes 45% of the sample and in which only 7% of the members churned on the bottom left. You are moved to the left leave if you have a monthly contract that incorporates 55% of the sample.
Here another variable is now used for the split: the type of internet connection. If you have fiber (>1.5), the tree sends you to the left leave, which consists of 25% of the sample; in case you have base DSL or DSL16 (<= 1.5), you are sent to the right leave, which consists of 30% of the sample.
Here platform age is used for another split. If you are longer than 15.5 weeks with the platform, you are assigned to the right leave consisting of 15% of the sample, which indicates that you are a churner. We must thus assume for every user who is more than 15.5 weeks with the platform, using a mid or slow DSL connection, and having a monthly payment plan, that this person is likely to churn and needs our attention. Checking the other bottom leaves, we can develop similar rules.
You may by now understand why decision trees are popular with managers. They are easy to understand, can be nicely visualized, and lead to actionable rules and insights.
We start our inspection at the root, where contract length is the first variable for a split. The tree shows that if you have a one- or two-year contract, you are immediately moved to the non-churning leave that takes 45% of the sample and in which only 7% of the members churned on the bottom left. You are moved to the right leave if you have a monthly contract that incorporates 55% of the sample.
Here another variable is now used for the split: the type of internet connection. If you have fiber, the tree sends you to the left leave, which consists of 25% of the sample; in case you have base DSL or DSL16, you are sent to the right leave, which consists of 30% of the sample.
Here platform age is used for another split. If you are longer than 16 weeks with the platform, you are assigned to the bottom leave consisting of 15% of the sample, which indicates that you are a churner. We must thus assume for every user who is more than 16 weeks with the platform, using a mid or slow DSL connection, and having a monthly payment plan, that this person is likely to churn and needs our attention. Checking the other bottom leaves, we can develop similar rules.
Check, e.g., the green leave with churners in the middle of the bottom row that incorporates 4% of the sample. Going up from here, we can similarly conclude that all people who pay each month more than 50 €, who use a base DSL connection, and are longer than five weeks with the platform while having a monthly subscription plan, are also likely to churn. You may by now understand why decision trees are popular with managers. They are easy to understand, can be nicely visualized, and lead to actionable rules and insights.
To finally assess the accuracy of the model and compare it to the Logistic Regression, we once again predict the churn column in the test data set. Don't let the code confuse you too much. While we use again the predict() function for the churn predictions, we need to change the prediction output from the prediction likelihoods to Yes and No votes, which then resemble the data in our train data set. So we can finally count how many times we correctly predicted the outcome of the variables and then use this information to calculate the tree's accuracy.
X_test = Logi_Data_encoded_test[['monthly_bill', 'gender_Male', 'millenial_Millenial', 'clanmembership_Yes',
'platformsince', 'avatar_Yes', 'internet_connect', 'extrapro_account_Yes',
'contract_length',
'customer_support_Yes', 'payment_type']]
y_test = Logi_Data_encoded_test['churn']
# Predict the target variable on the test set
predictions = my_1sttree_LogiTude.predict(X_test)
from sklearn.metrics import accuracy_score
# Calculate accuracy
accuracy = accuracy_score(y_test, predictions)
print(f"Decision Tree Accuracy: {accuracy}")
Decision Tree Accuracy: 0.7980068991950939
As you can see, the churn prediction accuracy is 79.80% and thus slightly under the accuracy of the logistic regression model.
Question: Why do you believe the decision tree model’s accuracy to be slightly below the accuracy of the logistic regression? What key differences between the two models can you identify?
To see if we can improve our single decision tree model, we now try a popular bagging extension of decision tree models: Random Forest. We use randomly sampled subsets to build different tree models, which we then combine to create a forest from the different trees.
For Random Forrests we again rely on another package that has been especially developed for random forrest applications.
The command for the random forest remains very similar to the previous commands. We first specify the depedent variable and seperate this variable from the independent variables with the ~ symbol. In addition we can specify how many random subsamples we want to create, i.e. how many different trees we want to plant into or forrest. Don't worry if the estimation takes a bit more time, given that we ask our computer to estimate 5000 different models, some patience is ceretainly in place!
from sklearn.ensemble import RandomForestClassifier
X_train_2 = Logi_Data_encoded_train[['monthly_bill' , 'gender_Male' ,'millenial_Millenial','clanmembership_Yes','platformsince', 'avatar_Yes', 'extrapro_account_Yes' , 'contract_length_One_year', 'contract_length_Two_year', 'customer_support_Yes', 'payment_type']]
y_train_2 = Logi_Data_encoded_train['churn_Yes']
X_test_2 = Logi_Data_encoded_test[['monthly_bill' , 'gender_Male' ,'millenial_Millenial','clanmembership_Yes','platformsince', 'avatar_Yes', 'extrapro_account_Yes' , 'contract_length_One_year', 'contract_length_Two_year', 'customer_support_Yes', 'payment_type']]
y_test_2 = Logi_Data_encoded_test['churn_Yes']
# Create the random forest classifier
my_1stforest_LogiTude = RandomForestClassifier(n_estimators=5000, random_state=42)
# Fit the model to the training data
my_1stforest_LogiTude.fit(X_train_2, y_train_2)
# Predict the target variable on the test set
predictions = my_1stforest_LogiTude.predict(X_test_2)
To compare the model’s accuracy with the single tree model and the logistic regression, we can further calculate the accuracy with the help of the test set and the following code.
# Calculate accuracy
accuracy = accuracy_score(y_test_2, predictions)
print(f"Random Forest Accuracy: {accuracy}")
Random Forest Accuracy: 0.9405902644691453
So we see that the Random Forrest indeed delivered the highest accuracy with a value of 94.05%
While the logistic regression allowed us to use the odd ratios, and the decision tree allowed us to examine the different branches, to gain an understanding of which factors increase churning likelihoods, the random forest package provides us with the feature importance scores to determine the features which have the strongest impact on churn.
from sklearn.inspection import permutation_importance
# Get feature importances
importances = my_1stforest_LogiTude.feature_importances_
std = np.std([tree.feature_importances_ for tree in my_1stforest_LogiTude.estimators_], axis=0)
indices = np.argsort(importances)[::-1]
# Plotting Mean Decrease in Gini Importance
plt.figure(figsize=(10, 6))
plt.title("Feature importances using Mean Decrease in Gini")
plt.bar(range(X_train_2.shape[1]), importances[indices], color="r", yerr=std[indices], align="center")
plt.xticks(range(X_train_2.shape[1]), X_train_2.columns[indices], rotation=90)
plt.xlim([-1, X_train_2.shape[1]])
plt.tight_layout()
plt.show()
# Now calculating permutation-based importance for Mean Decrease in Accuracy
result = permutation_importance(my_1stforest_LogiTude, X_test_2, y_test_2, n_repeats=10, random_state=42)
sorted_idx = result.importances_mean.argsort()
# Plotting Mean Decrease in Accuracy Importance
plt.figure(figsize=(10, 6))
plt.boxplot(result.importances[sorted_idx].T, vert=False, labels=X_test.columns[sorted_idx])
plt.title("Permutation Importance of the features (Mean Decrease in Accuracy)")
plt.tight_layout()
plt.show()
The first plot is the Gini coefficient. The higher the variable values here, the more important the variables are for the model. The second plot is the accuracy plot which shows how much worse the model would perform without the included variables. Thus, a large decrease (= high value on the y-axis) is associated with a variable of high estimation power.
Question: Which variables do have the strongest impact on churn? What can Logi.Tude learn from that and possibly improve?